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Abstract 

The paper considers the problem of estimating the parameters 
in a continuous time regression model with a non-Gaussian noise of 
pulse type. The noise is specified by the Ornstein-Uhlenbeck process 
driven by the mixture of a Brownian motion and a compound Poisson 
process. Improved estimates for the unknown regression parameters, 
based on a special modification of the James-Stein procedure with 
smaller quadratic risk than the usual least squares estimates, are pro- 
posed. The developed estimation scheme is applied for the improved 
parameter estimation in the discrete time regression with the autore- 
gressive noise depending on unknown nuisance parameters. 
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1 Introduction 



In 1961, James and Stein proposed shrinkage estimates which outperform 
in mean square accuracy the maximum likelihood estimates in the prob- 
lem of estimating the mean of a multidimensional Gaussian vector with unit 
covariance matrix (James and Stein 1961). This result stimulated the devel- 
opment of the theory of improved estimation for different regression models 
with dependent errors. Fourdrinier and Strawderman and Fourdrinier and 
Wells solved the problem of improved parametric estimation in the regres- 
sion with dependent non-Gaussian observations under spherically symmetric 
distributions of the noise (Fourdrinier and Strawderman 1996; Fourdrinier 
and Wells 1994). Fourdrinier and Pergamenshchikov and Konev and Perga- 
menchtchikov investigated the problem of improved estimation in nonpara- 
metric setting (Fourdrinier and Pergamenshchikov 2007; Konev and Perga- 
menchtchikov 2010). 

In this paper, we consider the problem of improved parametric estimation 
for a continuous time regression with dependent non-Gaussian noise of pulse 
type. The noise is specified by the Ornstein-Uhlenbeck process which is 
known to capture important distributional deviation from Gaussianity and 
to be appropriate for modelling various dependence structures (Barndorff- 
Nielsen and Shephard 2001). 

Consider a regression model satisfying the equation 



where 9 = (8i,...,8 p )' (the notation ' holds for transposition) is the vector 
of unknown parameters from a compact set C MP; {4>j{t))i<j< p are one- 
periodic [0, +oo) — >■ R functions, orthonormal in the space C 2 [0, 1]. The noise 
(£t)*>o m (jTJ) is assumed to be a non-Gaussian Ornstein-Uhlenbeck process 
obeying the following stochastic differential equation 



p 




(1) 



d£,t = a^ t dt + du, 



(2) 



where a < 0, (ut)t>o is a Levy process which is the mixture 



U t = QiW t + QiZ t 



(3) 
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of a standard Brownian motion (w t )t>o an d a compound Poisson process 
(z t ) t >o defined as 



where (N t ) t > is a Poisson process with the intensity A > 0, and (Yj)j>\ is 
a sequence of i.i.d. Gaussian random variables with parameters (0,1). The 
noise parameters a, Qi, Q2 and A are unknown. 

The problem is to construct improved estimates for the unknown vec- 
tor parameter on the basis of observations {yt)o<t<n, which have higher 
precision as compared with the least squares estimates (LSE). 

It will be observed that the regression model (pQ) is conditionally Gaussian 
given the cr-algebra Q = cr{N t ,t > 0} generated by the Poisson process. In 
Section [3] it is shown that the problem of estimating parameter in (TjQ) 
can be reduced to that of estimating the mean in a conditionally Gaussian 
distribution with a random covariance matrix depending on the unknown 
nuisance parameters. This enables one to construct a shrinkage estimates for 
unknown parameters (0\, . . . , p ) in ([I]). The main result is given in Theorem 
13.11 which claims that the proposed estimate has less risk than the LSE. 

In Section [2] we propose a special modification of James-Stein procedure 
for solving the problem of estimating the mean in a conditionally Gaussian 
distribution. This procedure allows one to control the mean square accuracy 
of estimates. It is shown (Theorem I2.ip that this estimate has less mean 
square risk than the usual LSE. 

The rest of the paper is organized as follows. In Section H] we apply Theo- 
rem [2J] to the problem of parameter estimation in a discrete time regression 
under a Gaussian autoregressive noise with unknown parameters. Appendix 
contains some technical results. 

2 On improved estimation in a conditionally 
Gaussian regression 

In Section [3J we will shown that the initial problem of estimating the param- 
eters (#i, . . . , 6p) in model (JTJ reduces to the following one. Suppose that the 
observation Y is a p-dimensional random vector which obeys the equation 




(4) 



Y = + Z 



(5) 
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where 9 is an unknown constant vector parameter from some compact set 
O C M p , £ is a conditionally Gaussian random vector with zero mean and 
the covariance matrix T>(Q), i.e. Law(^\Q) = Af p (0,V(Q)), where Q is some 
fixed cx-algebra. 

The problem is to estimate 9. 

Consider a shrinkage estimate for 9 of the form 



where c is a positive constant which will be specified later. 

The choice of this estimate fl6]) is motivated by the need to control the 
quadratic risk 



in the case of conditionally Gaussian model ([5]). 

It will be observed that such control can not be provided by the ordinary 
James-Stein estimate (obtained from ([6]) by the change of ||Y|| to ||Y|| 2 ). 

In order to get an explicit upper bound for the quadratic risk of estimate 
(J6j) we impose some conditions on the random covariance matrix T>{Q). 

Assume that 

(Ci) There exists a positive constant A*, such that the minimal eigenvalue 
of matrix T>(Q) satisfies the inequality 



(C2) The maximal eigenvalue of the matrix T>(Q) is bounded on some 
compact set C MP from above in the sense that 



where a* is a known positive constant. 

Further we will introduce some notation. Let denote the difference of the 




(6) 



R(9,9) = E fl ||0-0 



(V(Q)) > A* a.s. 



sup EgA max (V(Q)) < a*, 



risks of estimate (Q and LSE 9 = Y as 



A(0) := R(9*,9) - R(9,9). 



We will need also the following constant 



ES2¥(-i)p-v-w r (i+ 



) - {-^ym 



2p/ 2 - 1 r (1) d 
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where \i = d/\/a*, 

1(a) = I" exp( ~ r2/2 ^ r a nd d = sup{||0|| : 9 G 0}. 
Jo « + r 

Theorem 2.1. Let t/te noise £ in (jSJ) /icrae a conditionally Gaussian distribu- 
tion Ap(0, and ifo covariance matrix V(Q) satisfy conditions (Ci), (C 2 ) 
wit/i some compact set C M p . Tnen t/ie estimator (JB]) wii/i c = (p— 1)A*7 P 
dominates the LSE §ml for any p > 2, i.e. 

supA(0) < _[(p-l)A, 7p ] 2 . 

Proof. First we will find the lower bound for the random variable ||Y|| _1 . 
Lemma 2.2. Under the conditions of Theorem 2.1 

inf E e -J— > j p . 

eee \\Y\\ 

The proof of lemma is given in the Appendix. 

In order to obtain the upper bound for A(9) we will adjust the argument 
in the proof of Stein's lemma (James and Stein 1961) to the model flSJ) with 
a random covariance matrix. 

We represent the risks of LSE and of (J6j as 

R0, 9) = E e \\§ - 9\\ 2 = E e (E\\§ - 9\\ 2 \g) = E e trV(g); 
R(9\ 9) = R(9, 9) + E e [E((g(Y) - 1) 2 \\Y \\ 2 \Q)} 

+2^E e [E((o(F)- l)Y J (Y J -9 J )\g)}, 
i=i 

where g(Y) = 1 - c/\\Y\\. 

Denoting f(Y) = (g{Y) — l)Yj and applying the conditional density of 
distribution of a vector Y with respect to a-algebra Q 

, , r v 1 I (x - 9)'V-^g)(x - 0) 

M ' W = (2,)./Vde tP (g) eXP { 2 

one gets 

I j :=E(f(Y){Y j -O j )\G)= f f(x)(x-9 J )p Y (x\g)dx, j = l~p. 

Jrp 



Making the change of variable u = T> 1 ^ 2 {Q){x — 6) and assuming f{u) 
f{V l / 2 (Q)u + 0), one finds that 



h = 7^yf 2 Y,(^ 1/2 (G)hi £ /(«)«! exp f-JI^ ] du, j = l,p, 



\u\ 

' " (27T) 



where denotes the (i,j)-th element of matrix A. These quantities can 

be written as 



ij = EX>(< v'i\g) > il < vv\g) > kl ^L( u )\ u=Y \g), j = i~p. 
1=1 k=i OUk 

Thus, the risk for an estimator ([6]) takes the form 

R{9\6) = R(§,9) + E e ((g(Y) - 1) 2 \\Y\\ 2 ) 

+ 2E * (EEE < pl/2 (s) >n< 7)1/2 w >« - 

Therefore, one has 

R(6*,9) = R(6,6)+EgW(Y), 

where 

W(z) = c 2 + 2c ^|^ - 2trV ^ c J^y 

This implies that 

A(0) = EtfW(r). 
Since z'Az < A ma:r (/l)||2:|| 2 , one comes to the inequality 

2 trv(g)-x max (v(g)) 



A(0) < c 2 - 2cE 



||Y 

From here, it follows that 



A(0)<c 2 -2c^E e M^p. 



i=2 

Taking into account the condition (Ci) and the Lemma 12.21 one obtains 

,2 



A(0) < c 2 - 2(p - l)A*7 P c =: 0(c). 
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Minimizing the function 0(c) with respect to c, we come to the desired result, 
i.e. 

A(0)<-[(p-l)A* Tp f- 

Hence Theorem 12.11 

Corollary 2.3. Let in (jSJ) the noise £ ~ Af p (0,D) with the positive definite 
non random covariance matrix D > and X min (D) > A* > 0. Then the 
estimator fl6]) with c = (p — 1)A*7 P dominates the LSE for any p > 2 and 
compact set 6 C MP, i.e. 

supA(^) < -[(p-l)A, 7p ] 2 . 
eee 

Remark 2.1. Note that if D = a 2 I p then 

su V A(9)<-[(p-l)a\] 2 . 

Corollary 2.4. If £ ~ A/^(0, J p ) and 8 = in model ([5]) t/ien the risk of 
estimate ((6]) is given by the formula 



R(o,e*)= P - 



( P -i)T((p-i)/2y 2 



(7) 



v/2T(p/2) 

By applying the Stirling's formula for the Gamma function 

F(x) = V2^x x ~ 1/2 exp(-x) (1 + o(l)) 

one can check that r p — >■ 0.5 as p — )■ oo. The behavior of the risk for small 
values of p is shown in Fig.l. It will be observed that in this case the risk of 
the James-Stein estimate 9js remains constant for all p > 3, i.e. 

R(0jj S ) = 2 

and the risk of the LSE 9 is equal to p and tends to infinity as p — > oo. 



□ 
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Figure 1: Risk of 9* at 9 = 0. 

3 Improved estimation in a non-Gaussian Ornstein- 
Uhlenbeck— Levy regression model 

In this section we use the estimate (jEJ) to a non-Gaussian continuous time 
regression model to construct an improved estimate of the unknown vector 
parameter 9. To this end we reduce first the initial continuous time regression 
model 01) to a discrete time model of the form ((Sj) with a conditionally 
Gaussian noise. 

A commonly used estimator of an unknown vector 9 in model (0Q) on the 
basis of observations (yt)o<t<n is the LSE 9 = (9i, . . . , 9 P )' with the compo- 
nents 

1 f n 

6 j = - / wjdyt, j = hp- 

n Jo 

From here and (0Q), one has 

9 = 9 + n- 1/2 ({n), (8) 
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where ((n) is the random vector with the coordinates 



Note that the vector C( n ) has a conditionally Gaussian distribution with a 
zero mean and conditional covariance matrix V n (Q) = cov(((n), ((n)'\Q) with 
the elements 

Vij (n) = E(Ci(n)C-(«) I G)- 

Thus the initial problem of estimating parameter 9 in ([1]) can be reduced 
to the that of estimating parameter 9 in a conditionally Gaussian regression 
model ©. 

Theorem 3.1. Let the regression model be given by the equations ([I])-(J4]), 

Q\ > 0. Then, for any n > 1 and p > 2, the estimator of 9 



n\\9\\ 

dominates the LSE 9: 

~Qi(p- 1)7, 



0, 



sup A(0) < 

see 



n 



To prove this theorem one can apply Theorem 12.11 To this end it suf- 
fices to check conditions (Ci), (C2) on the matrix V n (Q). The verification of 
conditions (Ci) and (C2) is given in the Appendix. 



4 Improved estimation in an autoregression 

In this section we consider the problem of improved estimating the unknown 
mean of a multivariate normal distribution when the dispersion matrix is 
unknown and depends on some nuisance parameters. The models of au- 
toregressive type are widely used in time series analysis (Anderson 1994; 
Brockwell and Davis 1991). 

Let the noise £ = (£1, . . . , £ p )' in (JSJ) be described by a Gaussian autore- 
gression process 

£ fc = a£ fc _i + e fc , k = l,p, (9) 



9 



where \a\ < 1, E£ = and independent Gaussian (0,1) random 

variables. Assume that the parameter a in OH]) is unknown and belongs to 
interval [—a, a], where < a < 1 is known number. 

It is easy to check that the covariance of the noise £ has the form 



D(a) 



I -a 2 



( 1 a 
a 1 

7 P-1 n P~ 2 



7 p-2 



\ 



\ aP~ L aP~* ... 1 / 

Proposition 4.1. Let the noise £ in @ be specified by equation (J9]) with 
a G [—a, a]. Then, for any p > 1/(1 — a) 2 , the LSE is dominated by the 
estimate 

1 \ 7 P 



6* 



1 



P 



:i-a) 



\Y\ 



Y 



in the sense that 



sup A(0) < - [p 

flee 



'I -a) 2 1 7p ' 



Proof. We note that trD(a) = p/(l — a 2 ). Now we will estimate of the 
maximal eigenvalue of matrix D(a). From the definition 

Knax{D(a)) = sup z'D(a)z 
ll*[|=i 

one has 



p p 



z'D(a)z = J2J2 <D 
i=i j=i 



a) >ij ZiZj 



j / p-l p-i \ 

1 / p-i p-j 



ZjZi+j 



3=1 i=l 

Applying the Cauchy-Bunyakovskii inequality yields 



A, 



^))^( 1 + 2 S ai ) = (T^ 



a 



,2 ' 
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Thus, 

trD(a) - \ max {D{a)) >p- ' 



By applying Theorem 12.11 we come to the assertion of Proposition 14.11 



5 Appendix 

5.1 Proof of the Lemma [2^2 



Proof. From (loTh one has 



J = Efl— — = Eat— — > E ft 



Taking the repeated conditional expectation and noting that the random 
vector £ is conditionally Gaussian with zero mean, one gets 



(2n)p/ 2 v ^W{g) Jrp d+\\x 

Making the change of variable u = T>( K Q)~ l ' 2 x and applying the estimation 
u'V{Q)u < \ max (V(G))\\u\\\ we find 

r 1 f e Xp( -|HIV2) „ 



(2vr)p/ 2 7 RP d+^A max (P(6;))||«| 
Further making the spherical changes of the variables yields 

j > . i , ,e. r ir 



2 p/2 "T(p/2) °j„ i+yusf, 

From here by applying the Jensen and Cauchy-Bunyakovskii inequalities 
and by the condition (C2), we obtain 

/1 r r^ 1 exp(-r 2 /2) 

J - 2f/2-ir(p/2)d y ^+7 dr_7p - 

This leads to the assertion of Lemma 12.21 
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5.2 The verification of the conditions (Ci) and (C2) on 
the matrix V n {Q). 

Now we establish some properties of a stochastic integral 

W) = f /A < t < n 

with respect to the process <$fy. We will need some notations. Let us denote 

£ f (t)=a / exp{a(t — v)} f(v) (1 + exp(2av)) dv , 
Jo 

where / is [0, +00) — > K. function integrated on any finite interval. We 
introduce also the following transformation 

T U t ) = \f Q (y{s)g{s) + e) g {s)) ds 

of square integrable [0, +00) —> R functions / and g. Here 

el g (t) = f(t)e g (t)+e f (t)g(t). 
Proposition 5.1. Iff and g are functions from £ 2 [0,n] then 

EI t (f)I t (g) = g*r Lg (t) (10) 

where g* = g\ + \g\- 

Proof. Noting that the process I t (f) satisfies the stochastic equation 

dl t (f) = af{t)i t dt + f(t)du t , J (/) = , 
and applying the Ito formula, one obtains fflUj) . Hence Proposition 15. II 

□ 

Corollary 5.2. Iff is function from C 2 [0,n] then 



rn 

EP(f)<3g* / f(t)dt. (11) 
Jo 
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Further, for an integrated [0, +00) — > K. function /, we define the function 

Lj(x, z) = a exp(ax) ^f(z) + a J exp(at>) / (v + z) dv^j . 

Let (Tj)z>i be the jump times of the Poisson process (N t ) t > , i.e. 

Ti = inf{t >0:N t = l}. 

Proposition 5.3. Let f and g be bounded left- continuous [0, 00) x Q — >• M 

functions measurable with respect to B[0, +00) (^) Q (the product a algebra 
created by B[0, +00) and Q ). Then 

E(J 4 (/)|0) = O 

and 

e (/,(/) i t {g)\Q) = gir f>g (t) +gij2 mum^ 

1>1 

+ Ql E / U\s)L 9 (s - T h T{) + g(s)L f (s - T h 7])) l (T;<s) rfs. 
Proof. By the Ito formula one has 

W) I t {g) = f (Qlmg(s) + a(f(s)I s (g) + g(s)I s (f))Qds 

+ Q 2 2 J2f^9(T l )Y l H {Ti , t} 

i>i 

+ f\mi s -(g) + g(s)i s Mmu s . 

Jo 

Taking the conditional expectation E (-\Q), on the set {TJ > t}, yields 
E(I t (f)I t (g)\G) = f Qlf(s)g(s)d S + Ql £ /(Tjgft) l {Ti<t} 

+ a f (f( S )E(I s (g)Qg)+g( S )E(I s (f)Z s \g))d S . 
•Jo 
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It is easy to check that 

aE(i t (f)£ t \g) = | £/ (t) + el L A l - t p t j) 1 {T J <ty 

From here one comes to the desired equality. Hence Proposition 15.31 

□ 

Thus, in view of Cj( n ) = n ~ 1 ^ 2 In(<fij) and Proposition 15.31 the elements of 
covariance matrix V n (Q) can be written as 

Q 2 r 

Vij(n) = — (j)i(t)^j(t)dt 
n Jo 

2 pfi 2 
^° 1>1 

o 2 f n 

+ (^W^^-T^TO + ^WL^^-T^TOjl^^rft. (12) 

71 i>i Jo 

Lemma 5.4. Let (£,t)t>o be defined by ([2]) with a < 0. Then a matrix V n (Q) = 
(vij(4>))i<ij< p with elements defined by (TT2|) . satisfy the following inequality 
a.s. 

inf inf z'V n {Q)z > q\. 

n>l || je: || 

Proof. Notice that by (1T2T) one can the matrix V n (Q) present as 

V n {G) = gll p + F n + B n {g), 
where F n is non random matrix with elements 



fij(n) = ^J W*)^(*) + dt 
and B n (Q) is a random matrix with elements 



i>i 
+ 



rn 

/ (<pi(t)L^(t - T h Ti) + <f>j{t)L^t - T h T t )) l m <t)dt\. 
Jo 
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This implies that 

z'V n (Q)z = g\z'z + z'F n z + z'B n {Q)z > g\z'z, 

therefore 

inf z'V n (G)z > q\ 
lkll=i 

and we come to the assertion of Lemma 15.41 

□ 

Lemma 5.5. Let (C,t)t>o be defined by ([2]) with a < 0. Then a maximal 
eigenvalue of the matrix V n {Q) = (%(ra))i<t i: ,<p with elements defined by 
(|T2|) . satisfy the following inequality 

sup sup EgA maa; (V n (G)) < 3 P o*. 

n>\ 6t=e 

where g* = gf + \g%. 
Proof. We note that 

p p 
E X max (V n (g)) < E g tr(V n (g)) = J2 E eC-(n) = -^E^(^)- 

Applying (HTJ and 4>){t)dt = n, we obtain the desired inequality. 
Hence Lemma [5.51 

□ 

Thus the matrix V n (Q) is positive definite and satisfies for any compact 
set 9 C MP, the conditions (Ci) and (C2) with A* = g\ and a* = 3pg*. 
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